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ABSTRACT 

Information  about  data  collection  and  modeling  risks  are  frequently  locked  with  information  providers  ra¬ 
ther  than  shared  with  downstream  information  consumers.  Information  consumers  often  ingest  products 
automatically.  Without  protocols  to  inject  uncertainty,  the  ensemble  modeling  products  common  in  the 
modeling  discipline  cannot  accurately  account  for  the  input  uncertainty  inherent  to  those  products.  Future 
work  to  establish  use  cases  and  incoiporate  practitioner-driven  rules  and  protocols  for  transmitting  tiered 
uncertainty  information  between  information  product  producers  and  consumers  will  advance  the  needs  of 
environmental,  social,  and  economic  actors  in  the  ensemble  modeling  production  chain.  This  in  turn  will 
allow  for  improved  error  transmission  throughout  the  decision  making  enterprise. 

1  CRITICAL  NEED  IN  ENVIRONMENTAL  AND  SUSTAINABILITY  POLICY 

Environmental  and  sustainability  policy  is  driven  by  the  need  to  effectively  convey  uncertainty  informa¬ 
tion.  As  noted  by  Adams  (2006),  sustainability  is  ultimately  a  problem  that  must  transcend  environmen¬ 
tal,  social,  and  economic  boundaries.  In  addressing  the  work  by  Edwards  (2010),  Allen  (2010)  noted  that 
“We  should  aim  to  convert  unknown-unknowns  into  known-unknowns,  [and]  not  pretend  we  can  elimi¬ 
nate  them.”  To  this  end,  a  key  driver  in  communicating  environmental  data  products  is  the  effective 
communication  of  uncertainty  as  a  first -order  information  product.  When  there  is  a  failure  of  the  envi¬ 
ronmental  domain  to  properly  convey  uncertainty  to  the  social  and  economic  domains,  there  is  significant 
risk  for  loss  of  trust  (Anon.  2010).  At  the  same  time,  increasing  domain  specificity  amplifies  the  amount 
of  uncertainty  experienced  by  elements  of  these  groups  (Gross  2010).  Furthermore,  the  importance  of  un¬ 
derstanding  climate  change  has  been  identified  as  a  key  driver  for  national  security  policy  with  the  United 
States  Navy  (Committee  on  National  Security  Implications  of  Climate  Change  for  U.S.  Naval  Forces;  Na¬ 
tional  Research  Council  2011).  The  significance  of  the  impact  of  uncertainty  conveyance  is  reflected  in 
the  literature  where  numerous  climate  change  researchers  have  identified  the  communication  of  uncertain¬ 
ty  as  a  challenge,  including  Bitz  (2008). 
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The  practical  challenge  of  conveying  uncertainty  is  heightened  because  the  economic,  social,  and  en¬ 
vironmental  domains  do  not  always  share  a  common  understanding  of  uncertainty.  Economic  measures 
for  uncertainty  often  focus  on  risk.  For  example,  a  seminal  effort  by  Knight  (1921)  first  identified  the 
‘the  ones  we  don’t  know’,  the  type  of  risk  that  would  be  immortalized  as  the  ‘unknown-unknowns’  of 
Secretary  of  Defense  Donald  Rumsfeld  (U.S.  Department  of  Defense  2002).  Trivially,  any  effort  to 
communicate  uncertainty  cannot  communicate  these  risks,  but  can  only  motivate  our  downstream  target 
audience  to  include  the  risks  communicated  by  upstream  uncertainty.  Nobel  prize  winning  approaches  to 
the  applications  of  the  decisions  between  uncertain  alternatives  includes  Tversky  and  Kahneman  (1992). 
Hirshleifer  (1979)  conducts  an  excellent  survey  on  some  of  the  great  thinking  about  information  uncer¬ 
tainty  in  the  economics  domain,  including  discussion  about  modem  portfolio  theory  from  Markowitz 
(1959).  The  visceral  nature  of  uncertainty  as  applied  to  information  asymmetry  that  reducing  your  auto¬ 
mobile  trade-in  are  discussed  in  Akerlof  (1970). 

Social  models  for  uncertainty  certainly  appear  to  lag  the  efforts  in  the  environment  and  economic 
domains.  Examples  thought  to  be  well  solved  in  the  environmental  domain  such  as  near  term  weather 
forecast  still  show  cracks  in  conveying  uncertainty.  A  recent  effort  by  Morss,  Demuth,  and  Lazo  (2008) 
demonstrated  the  different  ways  that  the  general  public  interprets  the  implicit  and  explicit  uncertainty  in 
weather  forecasts.  The  book  by  Patterson  (2003)  discussed  the  challenges  faced  by  the  public  in  digesting 
uncertainty  in  public  policy. 

2  GEOSPATIAL  DOMAIN  EFFORTS 

The  Joint  Committee  for  Guides  in  Metrology  (JCGM)  publishes  standards  and  shares  these  standards 
with  partners  such  as  ISO,  1UPAC,  1LAC,  B1PM,  IEC,  1FCC,  OIML,  1UPAP.  Two  major  approaches  in¬ 
clude  the  Guide  to  the  Expression  of  uncertainty  in  measurement  (GUM)  (B1PM  et  al.,  2010)  and  the  as¬ 
sociated  document  on  propagation  of  distributions  using  a  Monte  Carlo  method  (B1PM  et  al.,  2008). 
Numerous  specific  papers  can  be  found  to  highlight  the  use  of  these  standards  in  practice  (Lira  and 
Grientschnig  2010;  Bich,  Cox,  and  Harris  2006)  or  even  teaching  the  communication  of  uncertainty. 
However,  it  is  difficult  to  say  whether  these  documents  offer  a  conclusive  solution  to  the  challenges  of 
communicating  uncertainty. 

An  exciting  effort  is  underway  under  the  Seventh  Framework  Programme  that  addresses  the  need  to 
communicate  uncertainty  by  establishing  a  markup  language  standard  called  UncertML  (Williams  et  al., 
2009)  for  the  Open  Geospatial  Consortium  (OGC),  the  leading  standards  organization  for  geospatial  ser¬ 
vices.  UncertML  is  intended  to  describe  unconditional  distributions  and  conditional  distributions  where 
the  conditioning  is  no  longer  of  interest,  e.g.  the  prediction  from  a  Bayesian  random  function.  UncertML 
is  part  of  a  broader  UncertWeb  project  which  is  tasked  with  making  “the  uncertainty  enabled  model  web  a 
reality”  (UncertWeb  2011).  This  includes  efforts  to  produce  APIs  to  support  information  and  service 
models  and  demonstrations  of  UncertWeb  concepts.  Despite  the  features  and  generalizations  provided  by 
UncertML,  it  tends  to  emphasize  random  variables  rather  than  a  range  of  probabilistic  models. 

Other  various  efforts  in  communicating  best  practices  in  uncertainty  as  applied  to  climate  modeling 
and  decision-making  have  also  been  documented,  including  Morgan  (2009).  Efforts  to  manage  uncertain¬ 
ty  across  the  1PCC  are  documented  in  Swart  et  al.  (2008)  and  the  broader  philosophical  challenges  to  a 
shared  uncertainty  lexicon  are  discussed  in  Manning  (2003).  DG  Robinson  (1998)  performs  an  extensive 
study  of  uncertainty  analysis  techniques  from  the  period  1956-1985  applicable  to  complex  systems. 

Regan,  Colyvan,  and  Burgman  (2002)  construct  an  ecologically  focused  taxonomy  of  uncertainty  in 
conservation  biology.  An  effort  by  Hunter  and  Goodchild  (1993)  identified  the  key  issues  of  definition, 
communication,  and  management  of  error  in  spatial  databases.  Goodchild  (1998)  takes  a  critical  look  at 
the  lack  of  uncertainty  information  in  G1S  systems.  Land  cover  change  modeling  in  Geographical  Infor¬ 
mation  Systems  (G1S)  based  on  historical  map  data  is  presented  in  presented  in  Leyk,  Boesch,  and  Weibel 
(2005).  Further  efforts  in  the  geospatial  domain  have  included  Thomson  et  al.  (2005),  Worboys  (1998), 
AM  MacEachren  et  al.  (2005),  Couclelis  (2003),  and  Torres  et  al.  (2004). 
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3  CHALLENGE  ACROSS  NUMEROUS  DOMAINS 

Transmitting  uncertainty  information  across  organizational  boundaries  exists  as  a  challenge  in  numerous 
contexts.  In  the  domains  of  intelligence  and  policy  analysis,  1ARPA  (Intelligence  Advanced  Research 
Projects  Activity)  has  targeted  the  program  ACE  (Aggregative  Contingent  Estimation)  to  solve  questions 
relating  to  the  elicitation,  aggregation,  and  communication  of  expert  opinion  used  to  forecast  global 
events.  Here  the  approach  is  to  apply  prediction  markets  to  collectively  assess  event  uncertainty,  as  pro¬ 
moted  in  Arrow  et  al.  (2008)  but  with  innovative  weighting  schemes  based  on  forecaster  traits  as  found  in 
Dani  et  al.  (2006),  Ranjan  and  Gneiting  (2010),  and  Cooke,  El  Saadany,  and  Huang  (2008).  Studies  such 
as  Tetlock  (2005)  establish  the  effectiveness  of  prediction  markets  as  well  as  unweighted  opinion  pools. 

Failures  in  communicating  model  uncertainty  have  been  linked  to  the  2008  United  States  financial 
crisis  (Salmon  2009).  Salmon  argues  that  the  inability  of  end  users  to  comprehend  the  model  (Li  2000) 
ultimately  left  the  financial  firms  vastly  underestimating  systemic  risk.  Writers  such  as  Taleb  (2008)  go 
so  far  to  assert  that  it  is  ultimately  impossible  to  statistically  communication  uncertainty  in  heavy-tailed 
situations,  an  argument  he  advances  more  carefully  in  Taleb  (2007).  Visual  representations  of  uncertainty 
in  intelligence  analysis  are  presented  in  Thomson  et  al.  (2005). 

While  presented  here  to  illustrate  common  problems,  intelligence  and  financial  analysis  are  not  the 
only  domains  where  the  transmission  of  uncertainty  is  relevant  and  critical.  Further,  environmental, 
geospatial,  and  other  domains  where  temporal  ensemble  models  are  common  tend  to  be  particularly  sus¬ 
ceptible  to  the  effects  of  uncertainty  conveyance/communication,  or  a  lack  thereof.  Much  of  the  literature 
in  these  domains  focus  on  communicating  uncertainty  in  a  descriptive,  qualitative  manner  rather  than  via 
analytic/quantitative  means  (Gigerenzer  et  al.,  2005;  Patt  and  Schrag  2003). 

4  CONCLUSION:  SIMULATION  ANALYSIS  EXPERIENCE  AND  PRACTIONER 
REALITY 

Simulation  practitioners  have  long  worked  on  the  boundary  between  rigorous  statistical  and  probabilistic 
means  of  representing  uncertainty  and  practitioners’  needs  to  make  reasonable  simplifications.  It  is  a  core 
competency  of  a  simulation  modeler’s  toolbox  to  both  elicit  uncertainty  through  input  modeling  tech¬ 
niques  and  communicate  uncertainty  through  output  analysis  techniques.  A  review  of  classic  simulation 
texts  such  as  Law  and  Kelton  (2000)  and  Schruben  (1995)  show  well  established  techniques  in  this  area. 
For  example,  one  approach  in  input  modeling  is  to  lead  a  subject  matter  expert  through  a  discussion  to 
characterize  the  broad  features  of  an  input  process  distribution  using  simple  distributions  such  as  the  tri¬ 
angular  or  beta  distributions. 

Practitioners  face  their  own  challenges  when  trying  to  incoiporate  uncertainty  analysis  in  their  work. 
Even  if  they  have  the  requisite  self-awareness  to  address  the  uncertainty  in  their  analysis  products,  the 
tools  available  may  be  difficult  to  apply.  Ultimately,  proper  uncertainty  protocols  must  address  the  chal¬ 
lenges  of  the  practitioner  balanced  against  the  technical  capabilities  of  the  mathematical  approach.  As  a 
demonstration  of  ability  for  statistical  distributions  to  describe  uncertainty,  consider  the  diversity  in  repre¬ 
sentations  of  a  univariate  random  variable  and  the  relationships  between  them  as  shown  Figure  1  from 
Leemis  and  McQueston  (2008).  For  a  statistician,  the  figure  is  beautiful  given  its  illustration  of  the  pos¬ 
sibilities  and  the  array  of  relationships,  but  a  practitioner  is  likely  to  be  challenged,  frozen,  or  befuddled 
by  the  paradox  of  choice  (Schwartz  2005).  On  the  one  hand,  a  theoretical  univariate  is  unlikely  to  clearly 
match  the  users  needs,  while  the  sheer  volume  of  options  will  create  a  cloud  of  possibilities  that  obfuscate 
appropriate  problem-specific  alternatives.  It  is  understandable  why  practitioners  tend  to  prefer  the  safety 
of  the  normal  distribution. 

When  complete,  this  work  will  support  current  and  future  efforts  to  add  certainty  measures  to  envi¬ 
ronmental  products.  By  providing  users  a  more  accurate  picture  of  the  certainty  of  data  values,  it  offers 
the  following  benefits:  enhanced  trustworthiness  of  decision  aid  data,  enabling  of  a  greater  degree  of  data 
sharing,  and  better  decisions  to  survey,  sample  or  measure  the  environment. 
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Figure  1:  Univariate  distribution  relationships  (Leemis  and  McQueston  2008) 
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